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ABSTRACT: The affordances of learning analytics (LA) dashboards and visualizations are 
being increasingly harnessed for enhancing 21 st century (21C) pedagogical and learning 
strategies and outcomes. However, use cases and empirical understandings of student 
experience, especially in the K-12 schooling sector and in Asian education contexts, remain 
relatively scarce in the field. Our paper addresses this knowledge gap in two ways. First, we 
present a first iteration design of a computer-supported collaborative critical reading and LA 
environment, 1/I//READ, and its 16-week implementation in a Singapore high school. Second, 
we foreground students' evaluative accounts of the benefits and drawbacks associated with 
l/l//READ's LA dashboard, which pointed to a number of potentialities and perils. Positives 
included 1) fostering greater self-awareness, reflective, and self-regulatory learning 
dispositions, 2) enhancing learning motivation and engagement, and 3) nurturing connective 
literacy among students. The motivational value of peer-referenced LA dashboard 
visualizations for stimulating healthy competition and game-like learning was identified 
alongside the perils of these serving to demoralize, pressurize, and trigger complacency in 
learners. This paper aims to shed light on the pedagogical complexities of designing LA that 
considers learners as a critical stakeholder group. 

Keywords: Learning analytics, computer-supported collaborative learning, critical literacy, 

21st century competences, dashboards 


1 INTRODUCTION 

1.1 LA Affordances in K-12 Learning Contexts 

The affordances of learning analytics (LA) dashboards and visualizations are being increasingly 
harnessed to enhance 21 st -century (21C) pedagogy, learning, and literacies. According to Verbert, 
Duval, Klerkx, Govaerts, and Santos (2013), having access to formative feedback from learning 
dashboards gives learners the opportunity for greater awareness and reflection on their learning, 
using the insights gained to modify their perceptions or behaviours. While LA dashboards are still a 
fairly nascent educational tool, the benefits of using LA tools have been highlighted in extant 
literature, with the use of LA dashboards being associated with better performance (Arnold & Pistilli, 
2012), greater opportunities for reflection, and increased motivation in students (Verbert et al., 
2013; Wise, Zhao, & Hausknecht, 2014). At the same time, studies indicate that the effectiveness of 
formative feedback is largely dependent on students' perceptions of the feedback received (Stiggins, 
2005; Wiliam, 2011), and that not all LA dashboards are useful for learning; some may even have 
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negative effects on student learning. For example, Beheshitha, Hatala, Gasevic, and Joksimovic 
(2016) reported that the quality of students' comments was positively associated with the use of 
quality-related dashboard visualizations but negatively associated with the use of class average 
dashboard visualizations for students high on self-avoidance goals. 

On this front, however, a much higher proportion of studies examining the pedagogical benefits and 
complexities associated with LA are set in the context of higher education and massive open online 
courses (MOOCs), as compared to research in K-12 settings. To illustrate, of approximately 790 
articles with LA as a keyword extracted from the Web of Science, EbscoHost, and ScienceDirect 
databases between 2011 and September 2016, 30.5% focused on higher education and 10% related 
to MOOCs, while a mere 32 papers (4%) were specific to K-12 learning contexts. Further, a clear lack 
of research has been noted on use-cases of learner dashboards in K-12 settings (Schwendimann et 
al., 2016). Consequently, research-informed empirical understandings of students' personal 
experiences with LA tools and environments at fostering 21C literacies remain scarce to date. This is 
especially so in the secondary schooling sector, and in Asian education contexts in particular (Ochoa, 
Suthers, Verbert, & Duval, 2014; Schwendimann et al., 2016). This paper addresses this knowledge 
gap by 1) presenting a first iteration design of a computer-supported collaborative critical reading 
and LA environment and its 16-week implementation in a Singapore high school with 15-year-old 
students; and 2) foregrounding these students' evaluative accounts and collective sense-making to 
draw out the pedagogical benefits and complexities of designing and implementing LA dashboards 
to foster 21C literacies and learning dispositions. 

1.2 LA and 21C Literacies 

A marked shift in contemporary understandings of literacy occurred in the mid-1980s when it was 
reconceptualized as a social practice inextricably linked to its context. Since then, notable works 
such as the New Literacy Studies (Gee, 1991, 2008; Street, 2003), Multimodal Literacy (Jewitt & 
Kress, 2003), Social Literacies (Street, 2014), and Multiliteracies (New London Group, 1996; Cope & 
Kalantzis, 2015) have been influential in redirecting views of literacy towards a plural ensemble of 
"literacies" — or literate behaviours that are "increasingly multiple, multimodal and mediated 
through new technology" (Burnett, Davies, Merchant, & Rowsell, 2014, p. 1). These theorizations of 
new literacies share a central commitment to the importance of developing in learners both the 
ability and disposition to critically frame, evaluate, understand, make-meaning, and use multiple 
forms of text in socially generative and productive ways across diverse communicative and learning 
contexts. This in turn encapsulates our conceptualization and use of the term 21C literacies in this 
paper. Following this view, critical reading development is taken as an essential component of strong 
language and literacy skills that are fundamental to young people's productive participation in the 
21C global knowledge economy. Despite its importance, however, it has been commonly observed 
by teachers in Singapore and internationally that students struggle with relating and responding to 
English language (EL) texts on a personal level, thereby failing to engage deeply and critically in 
reading tasks (Garcia, Mirra, Morrell, Martinez, & Scorza, 2015). This, in turn, has been attributed to 
two key factors. 
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First, texts used in EL lessons are still predominantly print-based (Cope & Kalantzis, 2015), even 
though today's youth now interact largely with both print and digital multimodal texts — that is, 
texts that incorporate two or more communication modes for meaning-making, such as verbal and 
written language, visual imagery, gesture, and sound (Jewitt & Kress, 2003; Walsh, 2010). Further, 
despite their continual engagement with rich and fluid social media platforms beyond school, 
students have limited opportunities in the conventional EL curriculum and pedagogical setup to 
extend their reading practices and interactions in socially, cognitively, and technologically engaging 
ways beyond the stipulated class time (Tan & McWilliam, 2009). A second and more pertinent 
reason is that students currently have very limited access to meaningful and timely formative 
feedback on various dimensions of their personal reading engagement and development progress 
(Davison, 2013), thereby significantly constraining their capacity to self-evaluate and positively 
modify their learning behaviours, even if they so desired. 

Until these issues are effectively addressed, students' low levels of EL engagement and weak critical 
reading skills are likely to persist, with adverse effects on their language proficiencies, and 21C 
capacities — critical thinking, collaboration, and communication. This challenge serves as the 
impetus for our design, implementation, and evaluation of I/1//READ, a web-based collaborative 
critical reading and LA environment aimed at fostering 21C literacies — that is, deepening critical 
reading engagement levels, promoting self-regulated and collaborative knowledge construction as a 
socially generative practice in the EL domain among Singapore Secondary 3 (Grade 9) students, 
during and beyond formal class time. 

2 W/READ: A COLLABORATIVE CRITICAL READING AND LA ENVIRONMENT 

I/I//READ was designed with the primary objective of motivating and scaffolding students to develop 
richer dialogue and quality interactions with peers around multimodal texts, thereby deepening 
their personal connection to and appreciation of collaborative and critical reading as a highly 
relevant, generative, and meaningful social practice. To achieve this, the techno-pedagogical design 
of I/I//READ focused on two key learning affordances: online peer interactions around reading, and 
the LA dashboard. 

2.1 Online Social Reading and Discussion Space 

An online social reading and discussion space was developed in I/I//READ. This learning affordance is 
underpinned by Vygotskian socio-constructivist theories, and informed by a multiliteracies 
pedagogical framework that foregrounds four key dimensions of effective contemporary literacy 
enculturation in learners: 1) situated practice, 2) overt instruction, 3) critical framing, and 4) 
transformed practice (Cope & Kalantzis, 2015; Tan & McWilliam, 2009). This online social reading 
and discussion space was designed for students to simultaneously view/read multimodal texts and 
collaboratively critique and discuss these texts with their peers, using a suite of intentionally 
designed micro-pedagogical scaffolding scripts and popovers (Figure 1). 

The micro-level of pedagogical scaffolding scripts comprised 7 critical lenses (Message, Purpose, 
Audience, Assumption, Viewpoint, Inference, Impact) and 5 critical talk types (Ideate: I think that...; 
Justify: I think so because...; Validate: I agree...; Challenge: I disagree...; Clarify: I need to ask...). Paul 
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and Elder's (2001) "wheel of critical reasoning" and our own work on dialogic indicators of collective 
creativity and criticality (Tan, Caleon, Jonathan, & Koh, 2014) informed the design of these critical 
lenses and talk types. Together, these served as a meta-cognitive schema for guiding students' 
collaborative critique of texts on I/1//READ, in that students are required to tag each of their 
comments/replies with one critical lens and one critical talk type. This is in line with past research that 
has advocated such functionality for personalized agentic learning (Beaudoin & Winne, 2009) and was 
found to help students develop perspective-taking and meaning-making capabilities (Eryilmaz, van der 
Pol, Ryan, Clark, & Mary, 2013). Furthermore, each critical lens and critical talk type tag contained a 
"popover" that provided students with question prompts and sentence starters. These served as a 
constant referential resource reminding students what each tag meant, and how these could be used 
to critique texts more deeply. 


Discuss! My Learning Dashboard Resources 


Text 1 "Our Education System' - 3R2 
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Figure 1: Collaborative critical reading and discussion of texts using pedagogical scaffolding scripts 

of critical lenses and collaborative talk types. 

A new multimodal text (incorporating a blend of written text, visual imagery including gestural 
modes, and sometimes video) that centred on a pertinent social, moral, and/or ethical dilemma was 
uploaded each week over a period of 16 weeks across two school semesters (Figure 2). In these 
weekly I/1//READ sessions, students had one 30-minute period during their formal EL curriculum time 
in school to read, comment, and reply to other classmates' posts on the texts. This weekly 30-minute 
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period constituted one-third of the weekly Secondary 3 EL curriculum time (3 x 35 minute periods 
per week for 16 weeks), and findings reported and discussed in subsequent sections should be 
interpreted in light of this level of exposure that students' had to 1/I//READ over the implementation 
period. This was the maximum formal curriculum time that the participating school leader and EL 
teachers collectively decided they were able to devote to this curriculum innovation without 
compromising other essential academic demands of the EL subject domain. Students were also 
encouraged to use 1/I//READ in their personal time beyond formal EL lessons. 


Let's Critique & Discuss! My Learning Dashboard Resources v (£) & Log Out 


Let’s Critique & Discuss! 



Figure 2: Multimodal texts co-developed by teachers and researchers (examples only). 


Architecturally, I/1//READ is built on WordPress, an open-source PHP website creation tool, with 
additional plugins and in-house programming codes to customize the functionalities for our 
purposes. One major plugin was BuddyPress, which enhanced the social learning aspect where users 
can join singular or multiple groups, and utilize interactive features including an activities feed, 
personal messaging, and a friendship system. As shown in Figure 3, all data are stored in a MySQL 
database, with structural modifications made to capture additional data required for the LA 
dashboard component of the learning environment. This is further explicated in the following 
section. 


WIREAD SYSTEM ARCHITECTURE 



Members 
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2.2 LA Dashboard 

The LA affordance of 1/1//READ was designed with the aspiration of providing rich, meaningful, and 
timely formative feedback to students and teachers throughout the term to help monitor reading 
engagement and progress, and modify learning strategies and pedagogical practices to improve 
outcomes. We focus here on the student LA dashboard, which consisted of four components 
designed to visualize a range of dispositional, discourse, and social network analytics (Ferguson & 
Buckingham Shum, 2012) alongside EL achievement data. We elaborate on each of the four I/I//READ 
LA components below: 


• My WiREAD Critique and Discussion Profile (Figure 4): discourse-related learning data on 
students' online reading engagement based on their frequency and length of comments and 
replies, as well as their usage of different critical lenses and critical talk types (described earlier in 
Section 2.1) across texts. 


WHAT IS MY LEVEL OF PARTICIPATION IN WIREAD? 
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Figure 4: My 1/1//READ Critique and Discussion Profile. 
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• My Learning Attitudes and 21C Skills Profile (Figure 5): dispositional learning data from 
student self-report questionnaires administered at the start and end of each school semester, 
using measures adapted from pre-validated scales of productive 21C learning dispositions 
(Tan, 2009; Tan & Nie, 2015). These included: five dimensions of 21C skills (critical thinking, 
creativity, curiosity, collaboration, open-mindedness); three dimensions of attitudes towards 
learning EL (self-efficacy, task value, and engagement); five dimensions of student-teacher 
relatedness (student-teacher communication, trust, alienation, autonomy, and competence 
support from the teacher); and four dimensions of learning goals and mindset (mastery and 
performance goal orientations; deep and surface learning). Examples of items for each 
dimension are shown in Appendix A. The students used a 7-point Likert response scale 
(l=strongly disagree to 7=strongly agree). The Cronbach alpha reliabilities associated with 
each dimension were satisfactory (.71-92). 


WHAT IS MY 21 st CENTURY SKILLS PROFILE? 

How imaginative, inventive, curious, and collaborative am I... 
How much of a critical thinker ami... 

How open am I to new ideas... 



WHAT ARE MY ATTITUDES TOWARDS LEARNING EL? 
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■ Me 
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■ Me 
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Communication 



How healthy is my learning mindset? 
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Me 
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Figure 5: My Learning Attitudes and 21C Skills Profile. 
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• My WiREAD Social Learning Network Map (Figure 6): sociograms reflecting students' position 
and influence within the I/I//READ learning network (ties based on students' online discussion 
of texts with others) and the class' learning network (ties based on students' nomination of 
who and how often they approached classmates to discuss texts critically). 


Cfeck the Generate button to show . 1 reset your WiREAD Discussion Network Map. 

Generate 


For anonymity, actual namea have been replaced. 



ESTHER 


The network map is based on you/your dassmates' replies to one another’s 
comments. 

Nodes of peope who have replied one another a'e connected by lines. 
Arrows point in the direction of replies. 

Thicker lines mean more replies between nodes. 

The more you post, the b gge- your node. 

The more you reply, the brighter your node. 

Your node will show as a grey square until you make a post. 

•• - 

KARAN 


4 

JERROD 


SHI QIAN 

JOSEPH 


ALVIN 


ZHEN HONG 


XINYI 


YANLIN 


KANISHA 


SATHISH 


FARE 

KENNETH 


Figure 6: My I/I//READ Social Learning Network Map. 


• My Reading Achievement (Figure 7): EL achievement data on students' reading grades on 

school-based assessments throughout the term captured in teachers' centralized information 
portal. 


My Visual Text score: 2/5 
My Narrative Text score: 10/20 
My Information Text score: 6/10 

Total Score: 18/35 
Percentage: 52.40% 



Visual Texts Narrative Texts Information Texts 

Total Score Types 


Figure 7: My Reading Achievement. 

The design of these four LA dashboard components were in turn developed through 1) a pilot term 
of iterative design-based research process involving cognitive labs and focus groups with a class of 
40 students and one EL Head of Department, and 2) informed by extant graphical visualizations of 
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dashboards such as the Student Activity Meter (Govaerts, Verbert, Duval, & Pardo, 2012) and others 
(Corrin & de Barba, 2014; Wise et al., 2014). 

Through this two-pronged pilot design and testing process, an intentional decision was made on 
pedagogical and research grounds to display the class average data as a form of social reference for 
students. This was, first, a practical decision urged by teacher-collaborators who leaned towards 
their professional experience that such peer/norm-referenced feedback had been observed to be 
useful in stimulating students' positive changes in learning behaviours. Second, this was a research 
intention to generate contextually nuanced empirical data to address an extant literature gap given 
the inconclusive findings in the tertiary education sector, along with a definitive lack of empirical 
studies in the secondary school sector on whether such peer/norm-referenced visualizations were 
helpful and/or harmful for student learning and in what ways (Corrin & de Barba, 2014; Wise et al., 
2014). The data indicators and corpuses for the LA dashboard visualizations were retrieved using 
PHP functions integrated into the WordPress framework. In this process, data was checked and 
sorted. Application algorithms were applied to the data, with processed data parsed to the 
visualization handler where the final data is presented to the end user in interactive visualized 
models. 

3 METHODS 

To develop more nuanced insights on the design and learning affordances of LA for optimizing 
learning, we ask, "How do students make sense of the benefits and drawbacks associated with 
l/l//READ's LA dashboard components and visualizations"? 

To address this question, we drew on a subset of data generated from our larger design-based quasi- 
experimental study that evaluated the impact of W/READ's collaborative critical reading and LA 
dashboard affordances on student learning outcomes (3 W/READ classes, N=116) as compared to a 
control group (3 classes, N=92) using a combination of pre/post-tests and self-reported questionnaires, 
and qualitative feedback forms and focus groups conducted at the start and end of the 16-week 
innovation term. The Secondary 3 (Grade 9, 15-year-old) student participants were generally 
comparable in terms of academic achievement (mid-upper academic ability track) and gender 
composition (52.9% female, 47.1% male). Given this paper's focus on understanding student 
perceptions of W/READ's LA dashboard, we draw specifically on data generated from I/I//READ classes 
(N=116 students) and report on findings of students' evaluative accounts of the LA dashboard 
components. These were gleaned from a combination of self-reported questionnaire scales measuring 
student perceptions of the LA dashboard's ease of use and usefulness, and their qualitative (textual) 
accounts expressed during student focus groups and in open-text feedback evaluation forms. 

In the questionnaire, students were asked to report their perceptions of the LA dashboard on two 
scales adapted from Tan (2009): 1) perceived ease of use (4 items) and 2) perceived usefulness (15 
items measuring four key productive learning dimensions: socialization, expression of identity and 
opinions, development of 21C skills and dispositions, and academic learning and performance), and a 
new scale on perceived helpfulness for learning and growth (four items). Examples of items for each of 
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these scales and their dimensions are provided in Table 1, along with their Cronbach's alpha (a) 
reliabilities, which were found to be satisfactory (.79-91). 


Table 1: Examples of Items of Perceived Ease of Use, Usefulness, and Helpfulness Scales 


Scale/Dimensions 

No. of 

Items 

Likert Scale 

a 

Example of Item 

Perceived ease of use 

4 

l=strongly 

.86 

WiREAD is easy to navigate. 

Perceived usefulness 


disagree to 


WiREAD can help me... 

- Socialization 

3 

7=strongly 

.91 

... feel more connected to my classmates 

- Express identity & opinions 

4 

agree 

.86 

... learn to voice and justify my opinions more 

- Develop 21C skills & 

5 


.86 

... develop creative thinking skills 

dispositions 





- Academic learning & 

3 


.79 

... improve my performance in EL tests 

performance 





Perceived helpfulness 

4 

l=not 

.81 

How helpful are the following WiREAD features 



helpful at all 


for your learning and growth? 



to 


1. My Achievement Data 



7=extremely 


2. My Learning Attitudes & 21C Skills 



helpful 


3. My WiREAD Discussion Data 





4. My Social Learning Network Profile maps 


In addition, student focus groups were conducted with 30 students (10 from each of the three I/1//READ 
classes) who were purposefully selected to reflect diverse users with a range of low to high levels of 
I/I//READ usage, reading achievement, and vocality in EL class. The remaining students in the I/1//READ 
classes who were not selected to participate in the student focus groups (N=86) completed a 
qualitative feedback form on the aspects of I/1//READ they found to be useful or otherwise for their 
learning and why. For this paper, our analysis focuses on students' textual accounts generated from 
their responses to the following questions asked in the student focus groups and the qualitative 
feedback forms: 


• Which of the learning dashboard data did you find useful for your learning, and why? 

• Which of the learning dashboard data did you find not useful for your learning, and why? 

• What improvements do you suggest for My Learning Data/Dashboard? 

The focus group discussions were audio-recorded and transcribed verbatim. Data was triangulated 
from the transcripts and the qualitative feedback forms and compiled into categories to address the 
research question. Two of the authors proceeded to identify patterns from the data and draw out 
larger themes according to thematic analytic strategies such as pattern matching advocated by Miles 
and Huberman (1994) and Yin (2009). 


4 RESULTS AND DISCUSSION 

4.1 Students Converged on LA Dashboard as Benefitting their Learning 

On the whole, students converged on the I/I//READ LA dashboard and its four constitutive elements 
as bearing multiple positive benefits that enhance their learning in the English language and literacy 
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domain as well as broader 21C literacies. We elaborate on these findings in the following sections, 
drawing from students' self-reported questionnaire data and their qualitative accounts of the LA 
dashboard components. 

Based on their questionnaire responses, students found the LA dashboard easy to use, navigate, and 
understand (M=5.54, SD=1.02, Figure 8). They were also generally positive about 1) each dashboard 
component's helpfulness to their overall learning and growth (Figure 9), and 2) the overall 
dashboard's usefulness for enhancing key dimensions of productive 21C learning — socialization, 
expression of identity and opinions, development of 21C skills and dispositions, and academic 
learning and performance (Oblinger & Oblinger, 2005; Tan, 2009; Turvey, 2006), as shown in Figure 
10 . 
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In terms of productive learning dimensions, the dashboard was perceived to be most useful for 
nurturing their expression of identity and opinions (M=4.95, SD=1.37), followed by developing their 
21C skills & dispositions (M=4.65, SD=1.24), then academic learning and performance (M=4.6, 
SD=1.28). Students perceived the LA dashboard to be least useful for bolstering their socialization in 
school (e.g., connectedness to classmates, expansion of learning network, M=4.46, SD=1.52) (Figure 
10 ). 
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Figure 10: Dashboard useful for enhancing key dimensions of productive 21C learning? 


For deeper insights into students' collective sense-making and "experienced realities" of the LA 
dashboard and its constitutive elements as benefitting their learning, we now turn the analytic lens 
to students' qualitative accounts. Our thematic analysis foregrounded three key learning benefits 
that students attributed to IV/READ's LA dashboard, particularly in terms of fostering a number of 
important 21C literacies. These are 1) fostering greater self-awareness, reflective, and self- 
regulatory learning dispositions, 2) enhancing learning motivation and engagement, and 3) 
cultivating connective literacy among students. 


4.1.1 Fostering greater self-awareness, reflective and self-regulatory learning dispositions 
Students recurrently articulated the usefulness of the LA dashboard in terms of its value for making 
visible their learning progress, dispositions, and behaviours, which they implied were largely vague, 
even invisible to them in their everyday schooling endeavors. Going further, students repeatedly 
described the LA dashboard and its various components as most informative in terms fostering 
greater levels of self-awareness and reflection of one's learning profile and progress, thereby 
prompting intentional goal-setting and self-regulatory adaptive learning strategies on their part. 


Interestingly, students frequently converged on the Learning Attitudes and 21C Skills Profile as the 
most "eye-opening" in helping them "discover," "realize," "understand," and “find out strengths and 
weaknesses" that they did not previously know about themselves: 

"I found that My Learning Attitudes and 21C Skills were the most useful. This dashboard 
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allowed me to realize what type of attitude I had towards learning English and the way that I 
was trying to achieve my EL goals. I've come to realize that my attitude towards EL was not 
the best as I only learnt for the sake of doing well in exams not for the sake of understanding 
the language. Thus looking at it, I am trying to change my learning habits and attitude 
towards English in a better and more positive direction" (Stl, 3R7). 

The learning benefit of this dispositional analytic was echoed in many other similar student 
accounts: 

"I found it very interesting and it helped me know more about how I learn, thus helping me 
adjust the way I learnt and studied. It also helped me understand 21st century skills and how 
much of each I displayed. Knowing my learning attitudes also helped me to understand 
where I am lacking and where I should work on." (Stl7, 3R6) 

These learner accounts reinforce the need for schools to place more overt emphasis on developing 
students 7 21C literacies and dispositions by appropriating the affordances of pedagogically 
meaningful and well-designed LA, not only because these serve as strong predictors of academic 
outcomes (Tempelaar, Rienties, & Giesbers, 2015), but even more so of longer-term employment 
and life outcomes (Levin, 2012). 

In a similar vein, students found the I/I//READ Critique and Discussion Profile to be beneficial in 
raising their self-awareness and self-reflection, thereby prompting intentional goal-setting and self- 
directed adaptive learning strategies to expand their critical literacy fluency: 

"I find that the Learning Attitudes and the WiREAD Discussion Data is useful. It helps me 
zoom in on where I am lacking and how I can further improve on the missing skills and 
technique I am not applying. It allows me to learn more on how to achieve an all-rounded 
answer in my future answers. It has so far proven effective!" (Stl4, 3R6) 

"I found My WiREAD Discussion Data useful as it shows me my strengths and weaknesses, 
this section let[s] me gauge my ability to answer comprehension questions. It also allowed 
me to estimate and know how much I am lacking and plan out what I need to do in order to 
improve." (St3, 3R6). Another classmate elaborated, "Learning Dashboard shows the graphs 
on the critical lens that one often uses. This will enable us to realize what we have not tried 
before and which critical lens we should try out more. This improves the exposure to the 
different critical lens and the more we practise on one critical lens, the more we are able to 
apply it." (St26, 3R6) 

Consistent accounts were also given by students of the Reading Achievement dashboard and the 
1/I//READ Social Learning Network Map fostering their reflective and self-regulatory resources as a 
naturally occurring element of their learning process: 

"[The Reading Achievement dashboard] was useful for my learning as it showed my progress 
for the various lessons and allowed me to evaluate and understand my learning progress." 

(Stl4, 3R1). Another peer in the class continued, "I also found the EL achievement data most 
useful ... as I was able to gauge how much progress for EL I have made as well as how I fair 
against other students ... through this, I know what I can work on in order to improve." (Stl9, 

3R1) 
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"My Social Learning Network Profile [was most useful]. It shows who you have responded to 
and vice versa. The thicker the lines means the more often discussion occurs between two 
people and those who have greatly contributed to discussions are placed nearer to the 
centre. I find it useful as it shows my progress and which direction I should be headed in 
order to get positioned nearer to the centre. It also shows me who I should have discussions 
with ... [that] will be more insightful and fruitful." (St2, 3R6) 

4.1.2 Enhancing learning motivation and engagement 

On a related note, frequently heard among the participant cohort were students' attributions of 
strong links between greater learner self-awareness and reflection as precursors of enhanced 
learning motivation and engagement — cognitive, affective, and behavioural (Wang & Eccles, 2012). 
For example, St8 shared: "It was useful. It allowed me to know what I'm at now, so that I can work 
harder to improve. So yah , / worked slightly harder and I got better for my [Semester Assessment] 2" 
(3R7-Line_217). Similar accounts were articulated in other classes: "My EL Achievement Data and 
WiREAD discussion data was the most useful. It shows where I stand in class, whether I'm 
underperforming or not. Hence, it motivates me to work harder and try new things , so that I can 
catch up with my classmates." (St8, 3R6) 

In particular, students described the I/I//READ Social Learning Network Map that visualized dynamic 
relational learning activity data as being especially engaging and motivating. Students frequently 
voiced comments such as the following: 

"shows me where I stand ... motivates me to work harder," "...made me more motivated to 
comment on other's answers" (Stl6, 3R7), "...it makes me more motivated to comment so 
that my [social network] dot can be bigger and brighter, and I will know who to look to for 
help" (Stl6, 3R7), "...motivate me to help or get help from my friends," "...serves as a 
reminder to write comments on texts we have not done. It also encourages me to be more 
active as I often visit this section to see my progress and effort" (St8, 3R7), "...it shows me 
the participation level of the class and see how active you are as compared to the class. I will 
know where I stand in the class and if it's below the average, like being at the edge of the 
network, I will be more determined to answer more questions and comment on my friend's 
answers, increasing my level of participation for the text." (St20, 3R6), and "...it's like to 
encourage yourself to extend ... the circle of people who you discuss English text with ... so 
it's like, to see your bubble grow and it's like, it is an incentive ... like discuss more with the 
whole class, instead of just your friends" (St2 3R1-Line 306 and 312) 

These student-user accounts concur with much of LA research on the potential of social 
transparency and "visible learning" for promoting self-regulated learning (Lockyer, Heathcote, & 
Dawson, 2013) and also that learner autonomy, competence, and relatedness are vital to intrinsic 
motivation and sustained behavioural change (Wise, 2014), following the research on self- 
determination theory by Deci and Ryan (2011). 

4.1.3 Cultivating "connective literacy ”— a nascent disposition among students 

In students' collective sense-making, also noteworthy was their recognition that the LA dashboard, 
especially its social network analytics, bore much value for nurturing a disposition we term 
reciprocal social learning, that is, a capacity to shift focus away from self-interest and self-sufficiency 
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to engage in productive help-giving and help-seeking learning behaviours. In students' words: 

"The spider web, some of our names, the bubbles were bigger, right? So it showed us like 
who can we find like if we ever need any help in English. And some of the people, whose 
bubbles are like smaller, they are out of the web thing, so we can like we can just ask them 
whether they need any help." (3R1, Line 270-285) 

"...it makes me more motivated to comment so that my [social network] dot can be bigger 
and brighter, and I will know who to look to for help... motivate me to help or get help from 
my friends."(Stl6, 3R7) 

This idea of reciprocal social learning brings us to the larger concept of connective literacy which we 
conceptualize and define as the capacity to see learning as an adaptive and fluid networked activity, 
where social positioning is seen as constituting a form of learning capital requiring active nurturing 
and reciprocal transactivity with significant others. In other words, connectively literate students 
recognize the importance of "seeing" their individual positioning within the larger class network, of 
knowing whom they were "connected with" and "stay[ing] connected" in ways that were profitable 
to learning, such as: 

1. To "find out the people that [they] can actually approach for English," to "know who [they] 
need to interact with more," to seek help from " prominent others" so as "to improve and 
learn from them" 

2. To "diversify" and "expand the circle of people whom [they] discuss English texts with... to 
take part in more conversations with the whole class, instead of just the people [they] know 
better as friends," thereby " mak[ing them] think more" or "forcing them to think deeper" 

3. To show consideration to "others at the side with hardly any connection... [to] try to see their 
comments more, and reply to their comments" 

On a more sobering note, however, we noted low levels of this "connective literacy" amongst the 
student participants. The views articulated above were noted to be relatively nascent and yet to 
emerge among many student participants. Instead, most students were heard to privilege individual 
learning outcomes and skill acquisition consistently as much more important than social learning 
connections: 

"To me, it doesn't matter who I'm connected to. As long as I can approach the text with 
different critical lenses, it is useful and helpful. My Social Learning Network Profile does not 
help me learn anything" (St8, 3R1). Others referred to the social learning network profile as a 
"redundant system," because "all it really did was show who worked with who more. It had 
no real significant purpose to help with my learning" (St29, 3R1), with yet others stating, "all 
I'm interested in is the content of the comments" (Stl, 3R6). 

Another student similarly added: 

"I found the part which showed me 'My Social Learning Network Profile' not useful. This is 
because I do not believe that there is a need to know about other people's progress or 
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whether or not they contribute to replying to others' comments or whether they ask 

questions, as long as I complete those tasks myself." (St3, 3R6) 

To many of these students, social networks have little bearing for learning and growth, with network 
centrality or prominence being trivialized as promoting a superfluous " social butterfly" syndrome, at 
the expense of more "serious learners... those who truly like to comment and learn" (StlO, 3R6). The 
inherently low levels of "connective literacy" amongst adolescent learners has implications for not 
only LA designers and pedagogues, but for a wider educational community committed to helping 
students become productive and engaged citizens beyond school. Though we acknowledge the need 
for a critical evaluation of l/l//READ's social learning network dashboard design and pedagogical 
value, these students' accounts reflect an emergent understanding of the power of social networks 
and how these extensively influence ideas, emotions, behaviours, learning, and more (Christakis & 
Fowler, 2009). This in turn brings to our attention an often-overlooked imperative — that educators 
of young leaners today need to be much more intentional and purposeful in efforts to develop an 
empirically informed appreciation for the power of adaptive and reciprocal social networked capital 
— as a form of new fundamental literacy essential to successful learning, living, and earning in 
contemporary societies. I/I//READ provides one such opportunity for building these literacies. 

4.2 Peer-Referenced Visualizations: A Double Edged Sword 

Students' qualitative accounts of the benefits gained from the LA dashboard, however, drew 
opposing counterpoints from other peers, thus providing insights into the ambivalence reflected in 
responses on the usefulness of the dashboard for their learning and growth. We highlight one key 
problematic here — that of the ironic nature of peer/norm-referenced versus self/criterion- 
referenced learning visualizations. 

4.2.1 Motivational Value of Peer-Referenced Visualizations: Healthy Competition, Game- 
Like Learning 

Students were polarized in their views of the norm-referenced visual analytics as indicators of 
academic and social standing among peers, and thus as a motivating mechanism for learning. For 
many, knowing "where they stand in class/' if they are "below the average" or "at the edge of the 
network" made them more "determined... to increase [their] level of participation," to "work 
harder," "be more active," and "improve" their learning behaviours online. Further, students often 
alluded to the inherent "fun" and "interesting" nature of the visualizations as stimulating a form of 
game-like learning (Gee, 2008) through "healthy peer pressure" and informal "competition" that 
drives them to engage with learning in more substantive ways: "the spider web would like, for me, it 
looks like a mini game, so that it's a race to get to the center and to get as big as you can" (Stl2, 
3R7). Higher-performing students in the classes even called for a more formalized "ranking system" 
or "hall of fame," asserting that this "leads to advancement and constant usage of WiREAD" (St25, 
3R7). A student explained: "...competitiveness is a way for you to improve actually... like if you... you 
want to always like beat the person that's the best one, right... so that you can be the best " (3R1- 
Line_752). The network visualization also allowed students to gauge their influence in the class 
discussion and "motivated" them to "become the most influential among" the class. 
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4.2.2 Perils of Peer-Referenced Visualizations: Demoralizing, Pressurizing, and Triggering 
Complacency 

There were, however, opposing views, and some pointed to the adverse affective impact of “clearly 
seeing" oneself in relation to others in the class, especially if one is “at the bottom " (My Reading 
Achievement), or “at the outside " (My Social Learning Network Map). A few students felt that it was 
demoralizing, inappropriate, and even discriminating to compare individual data with the class 
norm: 

"It was demoralizing... sometimes. You look like you are below average on the chart... then 
the class average is so high, and the maximum mark you are supposed to get is so high, then 
you are below average... it was demoralizing... depressing to see that classmates are better 
at commenting or thinking skills." (St5, 3R7) 

"...I do not like to compare my data with other people's data. This might cause some 
students to feel demoralized or depressed when they see that their classmates are better 
than them at commenting or in thinking skills." (St21, 3R1) 

"It shows... if one is not in the middle of web, it might seem as if he/she is out casted even 
though he/she could merely be a shy individual who is not vocal enough to express his/her 
thoughts. This can be seen as a type of discrimination if the person is offended. Those who 
are outside of the tangle of webs and links is apparent from the Network Profile, without a 
hint of subtlety to show how far and apart they are from the social circle." (St6, 3R1) 

A few also felt the dashboard provided additional unhealthy pressure for them: 

"After several classes, I have found out that working to a brighter and larger spot has turned 
into a competition ... this provides added stress that we do not need, as people have their 
own pace of learning and may be slower. The slow pace of learning may be laughed at by 
classmates and thus it is not as useful for learning as it may emotionally affect others." (Stl4, 

3R6) 

Besides the additional stress of having to keep up with the average, other students shared how they 
would just put in minimal effort to hit the average, and not go beyond. A student disclosed how he 
stopped himself from overly participating when he saw the dashboard: “...because after looking at it, 

I see that I comment too much... and then I started to comment less" (3R1-Line_708). Another 
student candidly shared: 

"Let's say, you just put the average... then people will just think that... I will just do one or 
two... cause that's the average... so that would mean that the student would not want to 
explore more." (3R1-Line_659) 

Students holding this view often asserted that learning visualizations are most helpful when they are 
criterion-based and self-referenced, rather than norm-referenced to peers: “seriously, I think what 
matters most is actually yourself instead of other people... it actually has some psychology effect. You 
shouldn't compare with anybody else cause even though this is collaborative learning, it's really more 
of self-learning in the end” (St3, 3R1); and “the purpose of comparing with others is to make you 
improve to so-called match other people's level, but comparing with yourself would, can actually 
have the same effect cause you are trying to improve based on your past. Comparing with yourself 
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will be more suitable. If like for example, the week before you commented less, then you think like 
'Oh yes, I actually commented more [this week], I feel better/ instead of comparing with others like 
'What? I'm last, what is this? Not fair' then you cry" (St8, 3R1). 

These polarizing views speak to the problematic nature of peer- and norm-referenced visualizations, 
which concur with findings from a small number of existing studies in the field of LA. In Wise et al.'s 
(2014) evaluation of the LA visualizations used in the E-Listening research program, they reported 
that although many students appreciated norm-referenced visualizations, some students found 
them discouraging and stressful, expressing a preference for criterion-based visualizations. Although 
this was in the context of higher education, the findings are similar to our findings from the K-12 
context. In addition, Corrin and de Barba (2014) reported that the use of class average data can 
induce a false sense of progress in students, with students being content with above-average 
performance even if it falls short of personal or course outcome targets. On a related note, the 
research conducted by Beheshitha et al. (2016) indicated that individual students' achievement goal 
orientations are associated with their perceptions of peer- and self-referenced LA visualizations, but 
the authors acknowledged that more in-depth research was needed on this front such that LA can 
be designed in ways that are adaptive to diverse learner needs and motivational profiles. 

The impact of norm-referenced and criterion-referenced feedback has its theoretical roots in 
assessment literature (Shute, 2008). Past studies indicate that when poor-performing students are 
presented with norm-referenced feedback, they tend to become less motivated and have lower self- 
efficacy, ascribing their performance to innate ability rather than effort (Chan & Lam, 2010; Kluger & 
DeNisi, 1996; Shute, 2008). Students' polarizing views on peer- and norm-referenced visualizations 
also foreground recent positive psychology understandings that emphasize "personal best" 
achievement goals as a highly salient predictor of students' academic motivation and engagement 
(Martin & Elliot, 2015). This perspective on "personal best" achievement goals is upheld by studies 
showing that the use of criterion-referenced feedback is associated with students ascribing their 
performance to effort (in turn reflecting a mastery-orientation) and expecting improvements in their 
performance (McColskey & Leary, 1985). Students' views on the shortcomings of peer-referenced 
visualizations reinforce the use of intrinsic motivation mechanisms in the design of LA to foster a 
mastery-oriented mindset that has been shown to result in more adaptive outcomes relative to a 
performance-oriented mindset (Dweck, 2012). 

5 CONCLUSION 

In this paper, student accounts of the LA dashboard revealed positive benefits to learning in terms of 
fostering greater self-awareness and self-regulatory learning dispositions, enhancing learning 
motivation and engagement, and nurturing connective literacy among students. At the same time, 
the "double-edged sword" nature of peer-referenced visualizations for stimulating competition, 
causing undue (felt) pressure, and triggering complacency in learners was foregrounded. These 
student perceptions arguably reveal different expectations of the EL activities and the techno- 
pedagogical design of I/I//READ. Wise, Vytasek, Hausknecht, and Zhao (2016) highlighted the 
pedagogical importance of helping students balance different reference points (e.g., peer average) 
and gain a better understanding of the values and limits of LA visualizations. In that regard, 
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HZ/READ's LA dashboard and pedagogical design could be buttressed with more scaffolding and 
prompts to help students engage in meta-cognitive self-regulatory learning behaviours in more 
meaningful and productive ways. 

To conclude, we highlight some limitations and ways forward. First, I/I//READ is only in its first design 
iteration and we are conscious of the LA dashboard's rudimentary visualizations. There is room for 
further enhancements to its user interface and aesthetic features. Second, we are working to 
incorporate richer and more proximal indicators and analytics, and in particular automated semantic 
and discourse analytics (Buckingham Shum & Ferguson, 2012; Rose et al., 2008; Rose & Ferschke, 
2016) that can better assess the quality of critical reading and thinking reflected in students' 
discursive practices. Relying on students' self-tagging of critical lenses and critical talk types leads to 
large variations in accuracy and may be of limited validity. For a start, we see potential to 
incorporate semi-automated tagging using existing dictionaries to identify students' EL learning and 
critical thinking skills at the individual level (Knight & Littleton, 2015). More time and scaffolding are 
needed to help students gain greater benefits from the interpretation of learning behaviour based 
on LA dashboard data. In future iterations, we will attempt to have a longer period for self- 
assessment and reflection on I/1//READ. The teacher dashboard is also being developed and refined to 
improve the adaptability of pedagogical strategies, giving consideration to learners' needs and 
interests. The aim is to enhance the pedagogical sensitivity and responsiveness of both the LA 
dashboard and the teacher-pedagogue in recognition of the tight coupling between extrinsic 
pedagogical scaffolding and intrinsic self-regulated learning (Azevedo, Moos, Greene, Winters, & 
Cromley, 2008). We also acknowledge that the I/I//READ intervention could benefit from more formal 
curriculum time being dedicated to it, which is a common struggle faced by many research-based 
curriculum innovation endeavors in the Singapore K-12 educational context. Having said this, 
I/I//READ is one of the first LA-focused learning environments trialled in the Singapore secondary 
schooling context involving 116 students and three EL teachers. At the time of writing, I/I//READ has 
been adopted by the incubator school across the whole year level, and trials are currently underway in 
three other secondary schools. 

The preliminary findings reported in this paper relate only to the first of three design, 
implementation, and evaluation cycles currently underway. Nevertheless, it is our hope that the 
data and discussion presented here fill some gaps in LA research by foregrounding secondary 
student-users' perceptions and experiences associated with the design and impact of LA for their 
overall learning and growth while shedding some light on relevant aspects of Asian edu-cultural 
contexts. This paper contributed empirical evidence on how students made sense of and accounted 
for the promise and perils of LA dashboards and visual analytics. We have also attempted to 
underscore some less discernible educational problematics that underlie LA design, relating to both 
instrumental and conceptual debates featured in the field's scholarly discourse. While definitely not 
a silver bullet, the affordances of the LA environment to make visible previously hidden learning 
behaviours, content, and interactions provide a possible means to motivate and engage learners in 
collaborative critical reading as a generative social practice that constitutes a fundamental element 
of productive 21C literacies. At the same time, the findings reported in this paper remind LA 
designers to be aware of the restrictive effects of one-sized-fits-all approaches to assessment and 
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pedagogy in conventional schooling (Gasevic, Dawson, Rogers, & Gasevic, 2016), and the need to 
develop richer, more nuanced and proximal multi-dimensional analytics (Dawson & Siemens, 2014). 
In this way, differentiated instruction can become an experienced reality for students, with 
purposefully designed LA serving to compress, rather than exacerbate, the learning and 
achievement gap between thriving and struggling students such that more meaningful and equitable 
educational experiences and outcomes may be realized. 
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APPENDIX A 

Examples of items for each dimension of “Productive 21C Learning 
Dispositions” 

(Visualized in l/WREAD student LA dashboard component "My Learning Attitudes and 21C Skills 
Profile") 


Dimension 

No. of 

items 

Cronbach's 

a 

Example of Item 

(Likert Scale of l=Strongly disagree to 7=Strongly Agree) 

Critical thinking 

3 

.86 

When a viewpoint is presented in EL doss, 1 analyze the 
supporting evidence to see if it is credible. 

Creativity 

4 

.88 

1 give imaginative (or novel) answers in EL 
discussions/tasks. 

Curiosity 

8 

.92 

1 like to ask questions when learning in EL lessons. 

Collaboration 

5 

.90 

1 try my best to contribute to group work in EL classes. 

Open-mindedness 

7 

.79 

1 am generally cautious about accepting new ideas. * 

EL self-efficacy 

5 

.88 

1 am sure 1 can learn the skills taught in EL subject well. 

EL task-value 

5 

.84 

What 1 learn in EL is useful. 

EL engagement: behavioural 

4 

.85 

1 work as hard as 1 can to learn EL. 

EL engagement: emotional 

4 

.86 

1 feel interested when we work on EL tasks. 

EL engagement: cognitive 

4 

.85 

1 make up my own examples to help me understand 
important concepts in EL. 

Student-teacher (S-T) 
relatedness: communication 

4 

.86 

1 tell my EL teacher about my achievements and failures. 

S-T relatedness: trust 

5 

.85 

My EL teacher believes in my capabilities. 

S-T relatedness: alienation 

3 

.71 

1 feel angry with my EL teacher. 

S-T relatedness: autonomy 
support 

4 

.85 

My EL teacher encourages me to ask guestions. 

S-T relatedness: competence 
support 

4 

.85 

My EL teacher gives me feedback that helps me improve 
my work. 

Mastery goals 

8 

.90 

The opportunity to do challenging work in EL class is 
important to me. 

Performance goals 

6 

.83 

1 like to be fairly confident that 1 can successfully perform 
an EL task before 1 try it. 

Surface learning 

10 

.88 

1 limit my study to what is needed for the EL exams as 1 
think there is no need to do anything extra. 

Deep learning 

10 

.90 

1 find new EL topics interesting and spend extra time 
trying to learn more about them. 


* reverse-coded item 
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